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FOREWORD 

Public  Law  93597,  the  Privacy  Act  of  1974,  includes  a requirement 
for  reporting  to  the  appropriate  authority  all  systems  of  records  from 
which  information  is  retrieved  by  name  or  other  identifying  particulars 
of  an  individual.  The  data  system  described  in  the  Procedure  Manual 
constitutes  such  a system.  However,  if  this  feedback  method  is  used  to 
fulfill  requirements  of  CNET  Instructions,  OPNAV  Instructions,  or  the 
BUPERS  Manual,  the  record  systems  will  be  described  centrally  and  the 
field  user  will  not  be  involved.  Any  question  about  whether  the  field 
user  of  this  feedback  method  or  a higher  command  is  responsible  for 
meeting  the  reporting  requirements  of  Public  Law  93597  should  be  directed 
to  higher  authority. 
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INTRODUCTION 

The  purpose  of  this  Procedure  Manual  is  to  provide  step-by-step 
instructions  and  other  aids  for  developing,  administering,  and  using  a 
feedback  questionnaire  which  will  enable  you  to  identify  training  problems 
that  may  exist  in  your  school.  These  procedures  were  developed  in  an 
extensive  study  of  Navy  training  feedback  methods.1  The  series  of 
activities  required  to  develop  and  use  this  questionnaire  is  shown 
schematically  in  figure  1.  Also  indicated  is  the  approximate  time  required 
to  complete  the  activities  in  each  block,  or  group  of  blocks.  The 
activities  in  blocks  I,  II,  and  III  occur  only  during  the  initial  feedback 
effort.  Feedback  activities  shown  in  the  other  blocks  will  recur 
periodically  as  long  as  the  training  course  exists. 

Each  section  of  this  Manual  corresponds  to  a block  in  figure  1. 

The  steps  presented  in  each  section  provide  detailed  instructions  for 
carrying  out  the  activity  in  that  block.  This  Manual  also  provides 
examples  of  forms  for  the  different  pages  of  the  questionnaire.  These 
can  be  reproduced  with  little  or  no  modification,  permitting  rapid 
questionnaire  development. 

Before  beginning  the  activity  in  the  first  block  of  figure  1,  it  is 
important  that  you  read  the  entire  Manual  and  familiarize  yourself  with 
all  of  the  different  feedback  activities.  It  is  also  important  to  read 
the  examples  of  questionnaire  forms  presented  in  figure  2 and  in  the 
appendices. 


Dyer,  Frederick  N.,  Ryan,  Leonard  E.,  and  Mew,  Dorothy  V.  A Method 
for  Obtaining  Post  Formal  Training  Feedback:  Development  and  Validation. 

TAEG  Report  No.  19.  1975.  Training  Analysis  and  Evaluation  Group, 

Orlando,  FL. 
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Figure  1.  Block  Diagram  of  Activities  Required  in 
Questionnaire  Development  and  Application 
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SECTION  I 

DEVELOPING  JOB  TASK  STATEMENTS 

Job  task  statements  form  the  "heart"  of  the  questionnaire,  and 
development  of  a complete  set  of  properly  written  statements  is  crucial 
for  a successful  feedback  effort.  If  a careful  job  task  analysis  in  the 
area  of  your  training  has  recently  been  conducted,  much  of  this  first 
activity  will  have  already  been  accomplished.  Development  of  these 
statements  is  accomplished  through  the  following  four  steps. 

STEP  1.  Obtain  Specific  Job  Task  Statements 

STEP  2.  Type  Statements  on  Cards 

STEP  3.  Review  Task  Statements 

STEP  4.  Revise  and  Add  Statements 

STEP  1.  Obtain  Specific  Job  Task  Statements.  The  definition  of  a job 
task  and  the  proper  form  for  writing  statements  to  describe  these  tasks 
are  presented  in  appendix  A.  If  the  job  for  which  your  school  provides 
training  has  had  a recent  task  analysis,  these  statements  will  already 
exist.  Obtain  a list  of  these  statements  from  curriculum  development 
personnel  or  from  other  sources.  Select  all  tasks  from  the  list  which 
presently  receive  training  emphasis  in  your  school  as  well  as  any  other 
tasks  performed  by  school  graduates  that  may  require  school  training, 
either  now  or  in  the  future. 

If  job  task  statements  are  not  available,  you  must  derive  them  from 
the  existing  course  curriculum  or  from  other  sources  such  as  technical 
manuals  and  instruction  books.  Follow  the  guidelines  for  doing  this  shown 
in  appendix  A. 

STEP  2.  Type  Statements  on  Cards.  Type  each  statement  obtained  in  Step 
1 on  a separate  3x5  card. 

STEP  3.  Review  Task  Statements.  Select  six  to  eight  personnel  who  are 
familiar  with  the  training  in  the  school  and/or  the  job  to  review  the 
task  statements.  These  could  include  instructors  and  curriculum  develop- 
ment personnel  and  should  include  at  least  two  recent  school  graduates. 

In  a face-to-face  interview  with  each  person,  review  each  task  statement 
and  discuss  whether  it  accurately  and  clearly  describes  the  job  task. 

Also  ask  these  personnel  if  any  additional  job  tasks  need  to  be  included 
in  the  questionnaire.  Continue  this  interview-edit  process  until  no 
further  changes  or  additions  are  recommended. 
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STEP  4.  Revise  and  Add  Statements.  Revise  task  statements  to  correct 
the  inaccuracies  and  ambiguities  identified  in  Step  3.  Prepare  additional 
task  statements  as  are  indicated  necessary  in  Step  3.  If  the  total 
number  of  task  statements  exceeds  200,  you  will  prepare  2 separate 
questionnaires,  each  with  half  of  the  task  statements.  This  is  described 
in  the  next  section. 


8 


TAEG  Report  No.  20 


SECTION  II 

PREPARING  THE  QUESTIONNAIRES 

Two  versions  of  the  questionnaire  are  to  be  prepared;  one  for 
trainees  and  one  for  supervisors.  Each  questionnaire  consists  of  five 
parts:  a cover  letter,  a biographical  data  sheet,  instructions,  task 

statements  and  associated  rating  scales,  and  an  open-ended  question  used 
to  obtain  additional  pertinent  information.  In  addition,  two  related 
letters  must  be  developed.  These  are  an  expectation  letter  to  alert  the 
receiver  and  a remi nder  letter  for  those  graduates  who  do  not  return 
the  questionnaire  on  time.  Preparation  of  these  materials  involves  the 
following  ten  steps. 

STEP  1.  Prepare  the  Cover  Letters 

STEP  2.  Prepare  the  Biographical  Data  Sheets 

STEP  3.  Prepare  the  Instructions 

A 

STEP  4.  Reproduce  the  Basic  Rating  Scale  Form 

STEP  5.  Type  Task  Statements  on  Rating  Scale  Forms 

STEP  6.  Prepare  the  Open-Ended  Question  Forms 

STEP  7.  Prepare  the  Expectation  Letter 

STEP  8.  Prepare  the  Reminder  Letter 

STEP  9.  Reproduce  Questionnaires  and  Related  Materials 

STEP  10.  Assemble  the  Questionnaire  Booklets 

STEP  1.  Prepare  the  Cover  Letters.  Page  1 of  the  questionnaire  will  be 
a cover  letter  addressed  to  the  person  who  will  receive  the  questionnaire. 
The  cover  letter  for  trainee  questionnaires  is  different  than  the  cover 
letter  for  supervisor  questionnaires.  Sample  letters  to  supervisors  and 
trainees  are  presented  in  appendixes  B and  C,  respectively.  These  may  be 
modified  to  meet  the  particular  needs  of  your  training  unit.  Type  these 
sheets  and  all  other  questionnaire  pages  parallel  to  the  long  dimension  of 
the  page  as  is  shown  in  the  examples. 

STEP  2.  Prepare  the  Biographical  Data  Sheets.  Page  2 of  the  questionnaire 
will  be  the  Biographical  Data  Sheet.  Prepare  one  Biographical  Data 
Sheet  for  supervisors  and  one  for  trainees.  Examples  are  shown  in 
appendixes  D and  E.  The  data  sheet  should  be  modified  from  these  examples 
to  suit  the  needs  of  your  training  unit.  For  example,  if  you  are  interested 
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in  looking  at  data  on  training  adequacy  separately  for  different  classes 
of  ships  or  different  educational  backgrounds  of  students,  include 
questions  which  will  provide  this  information  about  the  graduate. 

STEP  3.  Prepare  Instructions.  The  instuctions  will  be  on  pages  3 and  4 
of  the  questionnaire.  The  instructions  may  be  directly  reproduced  from 
the  examples  shown  in  appendixes  F and  G for  supervisors  and  trainees, 
respectively. 

STEP  4.  Reproduce  the  Basic  Rating  Scale  Form.  Reproduce  enough  copies 
of  the  form  shown  in  figure  2 to  accommodate  all  of  the  tasks  to  be 
included  in  the  questionnaire.  The  rating  form  includes  brief  instructions 
each  page  has  space  for  eight  job  task  statements  and  two  scales-- 
"Frequency  of  Task”  and  "Adequacy  of  School  Training  for  this  Task." 

Each  of  the  job  tasks  is  to  be  rated  by  circling  a number  from  one  to 
five.  If  optical  scoring  is  available  at  the  training  unit,  numbered, 
dotted,  parallel  lines  can  be  substituted  for  these  numbers.  An  addi- 
tional copy  of  this  form  for  reproduction  purposes  can  be  obtained  from 
the  Training  Appraisal  Branch  (N-34)  of  the  Chief  of  Naval  Education  and 
Training. 

STEP  5.  Type  Task  Statements  on  Rating  Scale  Forms.  Type  the  task 
statements,  described  in  section  I,  in  the  spaces  provided  at  the  left 
of  the  Basic  Rating  Scale  Form.  Precede  each  statement  with  a number, 
starting  with  number  "1."  Use  all  capital  letters  to  avoid  problems 
caused  by  the  need  to  capitalize  equipment  names.  Group  the  task  state- 
ments according  to  duties  (see  appendix  A).  Begin  the  questionnaire 
with  several  important,  frequently  performed  tasks  to  insure  that  the 
respondent  sees  the  relevance  of  the  questionnaire  to  his  job. 

STEP  6.  Prepare  Open-Ended  Question  Forms.  The  open-ended  question 
becomes  the  last  page  of  the  questionnaire.  You  may  reproduce  them 
directly  from  the  examples  in  appendix  H for  supervisors  and  appendix  I 
for  trainees. 

STEP  7.  Prepare  the  Expectation  Letter.  An  example  of  the  expectation 
letter  is  presented  in  appendix  J.  It  may  be  modified  to  suit  your 
particular  needs.  Do  not  prepare  an  expectation  letter  for  supervisors. 

STEP  8.  Prepare  the  Reminder  Letter.  An  example  of  the  reminder  letter 
is  shown  in  appendix  K.  It  may  be  modified  to  suit  your  particular 
needs.  Do  not  prepare  a reminder  letter  for  supervisors. 

STEP  9.  Reproduce  Questionnaires  and  Related  Materials.  Reproduce 
enough  questionnaire  pages  to  mail  to  the  selected  sample  (rules  for 
determining  the  number  of  trainees  and  supervisors  in  the  selected  sample 
are  provided  in  section  III).  Reproduce  some  extra  copies  to  be  used  if 
the  return  rate  is  insufficient.  Reproduce  as  many  expectation  letters 
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Figure  2.  Basic  Rating  Scale  Form 


11 


PAGE? 


FREQUENCY  OF  TASK 


Listed  below  are  tasks  which  presently  receive  at  least  some  emphasis  in  school. 
Please  rate  each  task  on  the  scales  at  the  right  by  circling  the  most  appropriate 
number.  Please  feel  free  to  also  include  your  reasons  for  your  rating  and/or  any 
specific  recommendations  for  training  on  this  task.  Your  comments Inay  be  written 
in  any  available  space  on  the  front  or  back  of  this  page  or  on  a separate  sheet. 


L Never  performed.* 

2.  Seldom  performed  or  only  In  emergencies. 

3.  Performed  monthly. 

4.  Performed  weekly. 

5.  Performed  dally. 


L 

2. 

3. 

A 

5. 


ADEQUACY  OF  SCHOOL  TRAINING 
FOR  THIS  TASK 


Task  requires  much  more  emphasis  In  school. 
Training  less  than  adequate  for  task, 

Increase  emphasis. 

Training  adequate  for  task. 

Training  more  than  adequate  for  task, 
reduce  emphasis. 

Greatly  reduce  or  eliminate  training  lor 
thl  s task. 


•ADEQUACY  scale  may  be  skipped  If  task  is 
never  performed. 


1 2 3 4 5 1 


1 2 3 4 5 1 


1 2 3 4 5 1 


1 2 3 4 5 1 


1 2 3 4 5 1 


1 2 3 4 5 1 


1 2 3 4 5 1 


1 2 3 4 5 1 


2 3 4 5 


2 3 4 5 


2 3 4 5 


2 3 4 5 


2 3 4 5 


2 3 4 5 


2 3 4 5 


2 3 4 5 
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as  trainee  questionnaires.  The  number  of  reminder  letters  to  be  reproduced 
is  70  percent  of  the  number  of  trainee  questionnaires.  Experience  has 
indicated  that  as  many  as  70  percent  of  the  trainees  will  need  to  be 
contacted  a second  time. 

STEP  10.  Assemble  the  Questionnaire  Booklets.  You  have  now  completed 
the  development  of  all  parts  of  the  questionnaire.  Assemble  the  pages 
in  the  order  shown  in  figure  3.  Staple  at  the  left  margin  of  the  pages  to 
make  a booklet. 

Situations  Requiring  More  than  One  Questionnaire.  With  more  than  200  task 
statements,  you  should  prepare  two  questionnaires  and  this  will  require 
twice  as  many  respondents  to  obtain  feedback  data.  Table  1 will  assist  you 
in  determining  the  proper  number  of  questionnaires  to  be  mailed.  Cover 
letters  should  indicate  that  only  half  of  the  job  tasks  are  included  on 
the  questionnaire.  Numbering  of  the  items  should  begin  with  "1"  on  both 
versions  so  that  the  person  receiving  the  questionnaire  does  not  believe 
he  has  received  incomplete  materials. 
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Open-Ended  Question 


Job  Task  Statements 


Job  Task  Statements 


Job  Task  Statements 


Instructions 


Biographical 
Data  Sheet 


Cover  Letter 


Figure  3.  Order  for  Assembling  Questionnaire  Pages 
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TABLE  1.  NUMBER  OF  TRAINEE  QUESTIONNAIRES  AND  SUPERVISOR  QUESTIONNAIRES  TO 
BE  MAILED  WITH  DIFFERENT  NUMBERS  OF  COURSE  GRADUATES 


Total  Number  of  Graduates  in  the  Past  Six  Months 

More  than  325  (650)* 

From  200  (400)  to  325  (650) 

Less  Than  200  (400) 

Randomly  select  200  (400)  to 

Randomly  select  125  (250)  from 

Each  graduate  receives  a form 

receive  trainee  forms. 

total.  The  remainder  receive 

and  his  supervisor  also  receives 

Randomly  select  another  125 
(250)  to  have  forms  mailed  to 
supervisors. 

Graduates  will  not  receive 
forms  when  forms  are  mailed 
to  their  supervisors. 

trainee  forms  and  no  forms  are 
mailed  to  their  supervisors. 

Of  the  originally  selected  125 
(250),  a number  equal  to  325 
(650)  minus  the  total  number  of 
graduates  receive  trainee  forms 
and  their  supervisors  also  re- 
ceive forms. 

All  others  have  forms  mailed  to 
their  supervisors,  but  do  not  re- 
ceive forms  themselves. 

a form. 

* Numbers  in  parentheses  are  for  situations  where  two  questionnaires  are  used  to  present  a very  large 
number  of  job  tasks. 
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SECTION  III 

SELECTION  OF  PERSONNEL  TO  RECEIVE  QUESTIONNAIRES 

You  are  now  ready  to  determine  the  number  of  questionnaires  to  mail 
and  also  the  graduates  (or  supervisors  of  graduates)  who  will  receive 
forms.  This  is  the  last  activity  to  be  completed  before  mailing  the 
questionnaires.  Only  the  first  five  steps  below  will  apply  in  most 
situations.  Step  6 is  to  be  followed  when  the  course  is  a very  long  one 
and  requires  two  questionnaires. 

STEP  1.  Obtain  Names  of  School  Graduates  from  the  Previous 
Six  Months'  Class  Rosters 

STEP  2.  Count  the  Number  of  Graduates  from  the  Previous  Six 
Months 

STEP  3 Selection  of  Personnel: 

a.  More  than  325  Graduates  in  Six  Months 

b.  Two  Hundred  to  325  Graduates  in  Six  Months 

c.  Less  than  200  Graduates  in  Six  Months 

STEP  4.  Obtain  Addresses  of  Duty  Stations  of  Selected  Personnel 

STEP  5.  Prepare  Lists  of  Personnel  Selected  to  Receive  Question- 
naires 

STEP  6.  Procedures  when  Using  Two  Questionnaires 

STEP  1.  Obtain  Names  of  School  Graduates  from  the  Previous  Six  Months' 
Class  Rosters.  Copy  graduation  rosters  of  personnel  who  have  completed 
your  school  in  the  last  six  months.  Obtain  the  Social  Security  numbers 
of  these  personnel  in  addition  to  their  names  for  use  in  randomly  select- 
ing a sample  of  these  graduates  in  Step  3 below.  It  is  important  to  use 
graduates  from  a full  six-month  period  even  if  a large  number  of  graduates 
would  be  available  in  a shorter  period. 

STEP  2.  Count  the  Number  of  Graduates  from  the  Previous  Six  Months.  If 
the  total  is  greater  than  325  then  you  will  mail  questionnaires  to  only 
a portion  of  the  available  graduates  (or  their  supervisors).  Follow  the 
procedures  of  Step  3a  for  random  selection  of  this  portion.  If  the 
total  is  325  or  less  but  more  than  200,  then  you  will  request  feedback 
data  on  each  graduate  with  either  the  graduate  receiving  a questionnaire, 
the  supervisor  of  the  graduate  receiving  a questionnaire,  or  in  some 
instances,  both  receiving  questionnaires.  These  procedures  are  described 
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in  Step  3b.  With  200  or  fewer  graduates,  all  graduates  and  their 
supervisors  will  be  involved.  These  procedures  are  described  in  Step  3c. 

STEP  3a.  Selection  of  Personnel  with  More  than  325  Graduates  in  Six 
Months.  This  procedure  is  summarized  in  the  first  column  of  table  1. 

Two  hundred  of  these  graduates  will  be  selected  randomly  from  the  total 
to  receive  trainee  questionnaires.  Another  125  will  be  randomly  selected 
to  have  supervisor  forms  mailed  to  their  supervisors.  Random  selection 
is  critical  to  insure  that  no  bias  related  to  duty  station,  class  standing 
sex,  etc.,  will  appear  in  the  sample. 

A simple  and  convenient  way  to  randomize  this  selection  is  to 
examine  the  Social  Security  number  of  each  graduate  and  select  on  the 
basis  of  the  last  two  digits  of  this  number.  To  do  this,  first  determine 
the  percentage  that  325  graduates  is  of  the  total  number  of  graduates  in 
the  last  six  months.  For  example,  if  there  were  750  graduates  available, 
the  325  would  be  43  percent  of  the  total.  By  selecting  graduates  whose 
Social  Security  numbers  end  in  01  through  43  (e.g.,  567-45-7301_,  453-77- 
6801  , 588-29-4402^,  ...),  approximately  325  graduates  will  be  obtained. 

If  the  actual  figure  is  less  than  325,  then  continue  to  add  graduates 
with  Social  Security  numbers  ending  in  numbers  beyond  43  until  325 
graduates  are  obtained.  If  the  number  of  graduates  with  Social  Security 
numbers  ending  in  the  digits  01  through  43  was  more  than  325,  then 
drop  Social  Security  numbers  ending  in  43,  42,  41,  etc.,  as  is  necessary, 
until  325  graduates  remain. 

A similar  procedure  can  be  used  to  randomly  select  the  125  graduates 
who  will  have  forms  mailed  to  their  supervisors.  Since  125  is  17  percent 
of  the  original  750  graduates,  those  graduates  with  Social  Security 
numbers  ending  in  01  through  17  will  closely  approximate  125  and  the 
exact  figure  could  be  obtained  by  adding  or  dropping  numbers  as  necessary. 
The  125  selected  would  not  receive  trainee  questionnaires,  but  their 
supervisors  would.  The  remaining  200  graduates  would  receive  trainee 
questionnaires. 

STEP  3b.  Selection  of  Personnel  with  200  to  325  Graduates  in  Six  Months. 
This  procedure  is  summarized  in  the  middle  column  of  table  1.  In  this 
case,  all  graduates  will  be  involved  in  the  feedback  effort  and  some 
will  receive  trainee  forms  and  also  have  forms  mailed  to  their  supervisors 
The  proportion  who  receive  trainee  forms,  the  proportion  who  receive 
supervisor  forms,  and  the  proportion  who  receive  trainee  forms  and  also 
have  forms  mailed  to  supervisors  must  be  selected  randomly  from  the 
total  number  of  graduates  available. 

The  randomization  procedure  based  on  Social  Security  numbers  can  be 
used  to  select  these  different  groups  of  graduates.  Supervisor  forms 
will  be  mailed  to  125  of  these  graduates.  Find  the  proportion  that  125 
is  of  the  total  number  of  graduates  available.  For  example,  if  there 
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were  275  graduates  to  choose  from,  125  would  be  45  percent  of  this 
total.  Those  graduates  with  the  last  two  digits  of  their  Social  Security 
numbers  between  01  and  45  would  closely  approximate  125.  Additional 
persons  could  be  added  or  dropped  as  discussed  in  Step  3a  until  exactly 
125  are  selected. 

The  remaining  graduates  will  receive  trainee  forms  and  will  not  have 
forms  mailed  to  their  supervisors.  However,  additional  graduates  will  be 
required  to  obtain  a total  of  200  graduates  who  receive  trainee  forms. 

This  additional  number  of  gradutes  must  be  selected  randomly  from  the 
group  of  125  who  will  have  forms  mailed  to  their  supervisor. 

In  the  example  with  275  graduates,  only  150  graduates  would  remain 
after  the  125  were  selected.  To  obtain  the  needed  total  of  200  graduates 
who  will  receive  trainee  forms,  50  graduates  must  be  randomly  selected 
from  the  125  graduates  whose  supervisors  will  receive  forms.  These  50 
would  be  18  percent  of  the  original  275.  Those  graduates  with  Social 
Security  numbers  ending  in  01  through  18  would  closely  approximate  the 
required  50. 

STEP  3c.  Selection  of  Personnel  with  Less  Than  200  Graduates  in  Six 
Months.  With  less  than  200  graduates  in  a six-month  period,  mail  a 
questionnaire  to  each  graduate  and  also  to  his  supervisor.  This  will 
produce  more  returned  questionnaires  for  supervisors  than  for  trainees, 
but  this  imbalance  is  better  than  not  approaching  the  150  returned 
questionnaires  that  are  needed  for  highly  accurate  rating  scale  data. 

If  the  school  has  a very  small  number  of  graduates,  extend  the  period  of 
graduate  selection  beyond  six  months  to  a year  or  longer  in  order  to 
build  up  a satisfactory  number  of  returned  questionnaires.  In  addition, 
use  more  intensive  follow-up  procedures  to  increase  the  return  of  question- 
naires. These  could  include  phone  calls,  mailing  of  a second  questionnaire, 
and  contacting  supervisors.  Even  if  all  efforts  produce  only  a small  total 
number  of  returned  questionnaires,  such  as  50  or  less,  the  data  should 
be  analyzed.  Much  caution,  however,  must  be  used  in  interpreting  the 
results,  since  chance  factors  could  produce  large  deviations  in  average 
ratings  with  such  small  amounts  of  data. 

STEP  4.  Obtain  Addresses  of  Duty  Stations  of  Selected  Personnel.  Obtain 
the  new  addresses  of  all  school  graduates  selected  in  Step  3.  These 
addresses  will  be  available  from  departure  orders  which  were  written  for 
these  personnel  at  the  time  of  graduation. 

STEP  5.  Prepare  Lists  of  Personnel  Selected  to  Receive  Questionnaires. 

Make  two  separate  lists  of  the  graduates  with  one  list  containing  the 
names  of  the  125  graduates  whose  supervisors  will  receive  questionnaires 
and  the  other  containing  the  names  of  the  200  graduates  who  receive  ques- 
tionnaires themselves.  Start  each  list  with  persons  who  graduated  six  months 
previously  and  end  with  the  most  current  graduates.  The  list  should 
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include  the  address  of  the  duty  station  of  the  graduate  and  the  date  of 
graduation. 

STEP  6.  Procedures  When  Using  Two  Questionnaires.  When  the  number  of 
job  task  statements  is  large  and  two  questionnaires  are  required  to 
obtain  rating  scale  data,  double  the  numbers  of  trainees  and  supervisors 
to  whom  questionnaires  are  mailed  to  400  and  250,  respectively.  Half  of 
each  group  will  receive  one  of  the  two  different  questionnaires  and  half 
will  receive  the  other.  Only  schools  with  large  enrollments  will  provide 
this  many  graduates  and  it  may  be  necessary  that  many  graduates  receive 
questionnaires  and  also  have  questionnaires  mailed  to  their  supervisors 
(see  parenthesized  figures  in  table  1).  If  this  still  does  not  provide 
sufficient  personnel,  use  the  alternatives  described  in  Step  3c  above  or 
reduce  the  number  of  task  statements  (e.g.,  by  combining  them)  in  order 
to  obtain  feedback  with  a single  questionnaire. 
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SECTION  IV 

MAILING  THE  QUESTIONNAIRE 

When  the  activities  of  personnel  selection  and  questionnaire  prepara- 
tion are  completed,  you  are  ready  to  mail  the  questionnaires.  This  activity 
requires  completion  of  the  following  five  steps: 

STEP  1.  Mail  Expectation  Letters  to  Trainees 

STEP  2.  Mail  Questionnaires  to  Trainees  and  Supervisors 

STEP  3.  Keep  Records  of  Mailing  Dates  and  Addressees 

STEP  4.  Mail  Reminder  Letters  to  Trainees 

STEP  5.  Follow-up 

STEP  1.  Mail  Expectation  Letters  to  Trainees.  Mail  the  expectation 
letter  directly  to  the  trainee  at  the  duty  station  indicated  on  his  depar- 
ture orders  or  to  a more  recent  address,  if  known.  Expectation  letters 
should  be  mailed  exactly  two  weeks  prior  to  mailing  the  questionnaire. 

Do  not  mail  expectation  letters  to  supervisors. 

STEP  2.  Mail  Questionnaires  to  Trainees  and  Supervisors.  Mail  the 
questionnaire  to  the  previously  selected  trainees  and  their  supervisors 
five  to  seven  months  after  the  trainee  has  graduated.  The  trainee  form  is  sent 
directly  to  the  trainee  at  his  duty  station  and  should  follow  the  expecta- 
tion letter  in  two  weeks.  The  supervisor  form  is  mailed  to  the  Commanding 
Officer  at  the  trainee's  duty  station  with  instructions  to  forward  the 
form  to  the  "Supervisor  most  qualified  to  evaluate  Seaman  (Name)." 

Include  a self-addressed  envelope  to  be  used  for  return  of  the  question- 
naire. Clip  or  staple  the  envelope  to  the  questionnaire  to  prevent  its 
loss. 

STEP  3.  Keep  Records  of  Mailing  Dates  and  Addresses.  Keep  accurate 
records  of  the  names  and  addresses  of  graduates  receiving  questionnaires 
and  of  graduates  whose  supervisors  receive  questionnaires.  The  two 
lists  of  graduates  compiled  in  Step  5 of  the  previous  section  can  be 
used  for  these  purposes.  Add  the  dates  of  mailing  of  expectation  letters 
(trainee  forms  only)  and  questionnaires.  Add  the  date  of  questionnaire 
return  to  these  records  as  well  as  the  Identification  Number  that  is 
assigned  to  the  returned  questionnaire  as  described  in  Step  2 of  Section  V. 
These  records  will  indicate  when  a reminder  letter  is  to  be  mailed  to 
trainees  who  do  not  return  questionnaires.  Add  the  date  of  mailing  of 
the  reminder  letter  to  these  records. 
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STEP  4.  Mail  Reminder  Letters  to  Trainees.  If  a trainee  has  not  returned 
his  questionnaire  within  one  month,  mail  one  of  the  reminder  letters 
prepared  for  trainees.  Send  the  letter  directly  to  the  graduate  and  not 
to  his  Commanding  Officer.  A reminder  letter  should  not  be  mailed  to 
the  supervisor  who  does  not  return  his  questionnaire. 

STEP  5.  Follow-Up.  After  questionnaires  and  reminder  letters  are 
mailed,  you  should  receive  an  average  of  three  trainee  forms  and  three 
supervisor  forms  returned  each  week.  If  after  two  or  three  months,  the 
average  rate  of  questionnaire  return  has  not  been  three  per  week  for 
both  trainees  and  supervisors,  mail  additional  questionnaires.  Select  the 
additional  personnel  randomly  from  the  total  number  of  graduates  for  the 
six-month  period  (except  those  previously  selected)  and  mail  questionnaires 
or,  for  trainee  forms,  mail  expectation  letters,  then  questionnaires, 
to  all  those  who  have  been  out  of  school  six  months  or  more. 

If  additional  graduates  are  unavailable  to  make  up  the  deficit  in 
returned  questionnaires,  you  should  then  use  more  intensive  efforts  to 
obtain  the  return  of  questionnaires  already  mailed.  These  could  include 
mailing  a second  questionnaire  and  contacting  graduates  or  their  super- 
visors by  phone. 
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SECTION  V 

RECORDING  DATA  FROM  RETURNED  QUESTIONNAIRES 

This  activity  should  begin  as  soon  as  the  first  questionnaires  are 
mailed  and  continue  for  a month  or  more  after  the  last  questionnaire  is 
mailed.  All  data  will  be  transferred  from  the  questionnaires  to  special 
data  sheets  which  facilitate  the  analysis  of  data  (data  analysis  is 
described  in  the  next  section).  Preparing  a notebook  of  data  sheets  and 
recording  the  data  from  rating  scales  and  open-ended  questions  are 
accomplished  via  the  following  five  steps. 

STEP  1.  Prepare  a Loose-Leaf  Notebook  with  a Separate  Page 
for  Each  Job  Task  Statement 

STEP  2.  Assign  Identification  (ID)  Numbers  to  Returned 
Questionnaires 

STEP  3.  Record  Questionnaire  Item  Ratings  by  Assigning  ID 
Number  to  Proper  Column  of  Item  Page 

STEP  4.  Write  Comments  for  an  Item  on  the  Back  of  the  Item 
Page  and  Include  ID  Number 

STEP  5.  Add  a Page  to  the  Notebook  for  Each  New  Item  Recom- 
mended for  Addition  to  the  Curriculum 

STEP  1.  Prepare  a Loose-Leaf  Notebook  with  a Separate  Page  for  Each  Job 
Task  Statement.  While  waiting  for  the  first  questionnaires  to  be  returned, 
prepare  a notebook  with  separate  pages  for  each  job  task  statement  on 
the  questionnaire.  Each  page  of  the  notebook  should  look  like  the 
Sample  Item  Summary  Data  Sheet  shown  in  figure  4.  Notice  that  space  is 
provided  for  keeping  separate  records  for  trainees  and  supervisors  for 
each  of  the  two  scales  of  the  questionnaire.  Write  the  numbered  job 
task  statement  at  the  top  of  the  page  and  arrange  the  pages  of  the 
notebook  so  they  correspond  to  the  order  of  items  on  the  questionnaire. 

STEP  2.  Assign  Identification  (ID)  Numbers  to  Returned  Questionnaires. 
Assign  an  identification  number  to  returned  questionnaires  and  cross- 
reference  this  number  to  the  record  of  names,  addresses  and  mailing 
dates.  Use  numbers  that  at  least  reflect  the  order  of  return  and  the 
trainee  (T)  or  supervisor  (S)  origin  (T1 , T2,  ...;  SI,  S2,  ...).  These 
numbers  should  also  reflect  other  important  information  which  may  require 
separate  data  analysis  such  as  boat  (B)  or  land  (L)  duty  station  (TB1 , 

TB2,  ...;  SB1 , SB2,  ...;  TL1 , TL2,  ...;  SL1 , SL2,  ...). 

STEP  3.  Record  Questionnaire  Item  Ratings  by  Assigning  ID  Number  to 
Proper  Column  of  Item  Page.  The  Sample  Item  Summary  Data  Sheet  in 
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ITEM  I 52 
TRAINEE 

Frequency  of  Task 


Use  principles  of  sky  wave  propagation  to  establish  a long  distance 
communication  path. 


Adequacy  of  School  Training 
For  This  Task 


SUPERVISOR 
Frequency  of  Task 


SUPERVISOR 

Adequacy  of  School  Training 
For  This  Task 


1.  Never  performed. 

2.  Seldom  performed  or 
only  in  emergencies. 

3.  Performed  monthly. 

4.  Performed  weekly. 

5.  Performed  daily. 


1 . Task  requires  much  more 
emphasis. 

2.  Training  less  than 
adequate  for  task, 
increase  emphasis. 

3.  Training  adequate  for 
task. 

4.  Training  sore  than  ade- 
quate for  task,  reduce 
emphasis. 

5.  Greatly  reduce  or  elimi- 
nate training  for  this 
task. 


1.  Never  performed. 

2.  Seldom  performed  or  only 
in  emergencies . 

3 . Performed  monthly . 

4.  Performed  weekly. 

5.  Performed  daily. 


2. 


5. 


Task  requires  much 
more  emphasis. 

Training  less  than  ade- 
quate for  task,  increase 
emphasis. 

Training  adequate  for  task. 
Training  more  than  ade- 
quate for  task,  reduce 
emphasis . 

Greatly  reduce  or  elimi- 
nate training  for  this 
task. 


Tl 

T3 

TS 


3 

T4 


i 


Tl 


T3 

T* 

TS 


i 

S2 

sV 


S3 


3 

SI 


SI 

S3 


5 

S'/ 


Figure  4.  Sample  Item  Summary  Data  Sheet 
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figure  4 illustrates  how  ratings  for  an  item  on  a returned  questionnaire 
are  recorded.  Write  the  ID  number  of  the  questionnaire  in  the  column 
corresponding  to  the  number  circled  by  the  trainee  or  supervisor  for  that 
item.  If  ratings  are  made  on  both  "Frequency  of  Task"  and  "Adequacy  of 
Training  for  this  Task,"  then  the  ID  number  will  be  written  twice  on 
the  notebook  page  corresponding  to  this  item.  With  100  items  on  the 
questionnaire,  the  ID  number  would  be  written  200  times  (less  if  Training 
Adequacy  scale  is  skipped)  in  recording  data  from  the  questionnaire. 

STEP  4.  Write  Comments  for  an  Item  on  the  Back  of  the  Item  Page  and 
Include  ID  Number.  Both  the  instructions  and  the  cover  letter  encourage 
the  respondent  to  make  written  comments  regarding  the  reasons  for  their 
rating  of  an  item,  recommendations  for  training  of  the  task,  or  any 
training  problems  for  a questionnaire  item.  When  such  comments  are 
provided,  write  them  on  the  back  of  the  notebook  page  for  the  item. 
Include  the  ID  number  of  the  questionnaire  with  each  such  comment.  If 
comments  are  long  and  reproduction  equipment  is  available,  you  may 
attach  a copy  of  the  comment  instead  of  providing  a time  consuming 
rewrite  of  it. 

STEP  5.  Add  a Page  to  the  Notebook  for  Each  New  Item  Recommended  for 
Addition  to  the  Curriculum.  When  a returned  questionnaire  has  a response 
to  the  open-ended  question  asking  for  new  material  that  should  be  added 
to  the  course,  add  a new  page  to  the  notebook  on  which  the  recommended 
item  is  written  along  with  the  ID  number  of  the  author.  As  future 
questionnaires  are  returned  which  mention  the  same  item,  add  their  ID 
numbers  to  this  page.  It  is  important  that  a separate  page  be  added  for 
each  new  item  that  is  mentioned. 
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SECTION  VI 

ANALYSIS  OF  QUESTIONNAIRE  DATA 

When  data  from  all  returned  questionnaires  are  recorded,  you  are 
ready  to  analyze  the  questionnaire  data.  The  analysis  will  summarize 
the  item  ratings  of  respondents  and  provide  overall  averages  which  will 
help  in  the  accurate  identification  of  areas  of  training  that  require 
change.  The  following  four  steps  are  necessary  to  complete  the  data 
analysis: 

STEP  1.  Calculate  Trainee  and  Supervisor  Averages  of  "Task 
Frequency"  for  Each  Item 

STEP  2.  Calculate  Trainee  and  Supervisor  Averages  of  "Training 
Adequacy"  for  Each  Item 

STEP  3.  Calculate  "Overall  Average  Training  Adequacy"  for 
Each  Item 

STEP  d.  Calculate  "Average  Training  Adequacy  for  All  Items" 

STEP  1.  Calculate  Trainee  and  Supervisor  Averages  of  "Task  Frequency" 
for  Each  Item.  Calculate  separate  averages  of  Frequency  of  Task  ratings 
for  trainees  and  for  supervisors.  These  means  can  be  calculated  quickly 
by  counting  the  number  of  persons  who  used  each  of  the  rating  categories, 
multiplying  each  rating  value  by  this  number,  and  dividing  the  sum  of 
these  products  by  the  total  number  of  ratings.  For  example,  if  5 persons 
rated  an  item  as  Never  Performed  (value  1),  10  persons  rated  it  as 
Seldom  Performed  (value  2),  25  persons  rated  it  Performed  Monthly 
(value  3),  25  persons  rated  it  Performed  Weekly  (value  4),  and  2 persons 
rated  it  Performed  Daily  (value  5);  the  products  of  rating  values  and 
number  making  the  rating  would  be  5,  20,  75,  100,  and  10,  respectively. 

The  sum  of  these  products  is  210  and  the  total  number  of  ratings  is  67. 
Dividing  210  by  67  gives  3.13  for  the  average  of  Task  Frequency.  Write 
these  trainee  and  supervisor  averages  of  Frequency  of  Task  ratings  at 
the  top  of  the  notebook  page  for  the  item. 

STEP  2.  Calculate  Trainee  and  Supervisor  Averages  of  "Training  Adequacy" 
for  Each  Item.  Follow  the  same  procedure  used  to  calculate  average 
Frequency  of  Task  to  calculate  the  averages  of  Training  Adequacy.  The 
number  of  persons  making  this  rating  will  usually  be  less  than  those 
making  Frequency  ratings.  It  is  important  to  use  the  actual  number 
of  ratings  as  the  denominator  in  calculating  the  average  rating  of 
Adequacy  of  Training  and  this  may  differ  from  the  number  of  returned 
questionnaires.  Write  these  two  averages  (one  for  trainees  and  one  for 
supervisors)  at  the  top  of  the  page  for  the  item. 
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For  both  Frequency  of  Task  and  Adequacy  of  Training  for  Task, 
additional  rating  averages  can  be  calculated  for  different  types  of  duty 
station  or  for  other  factors  that  are  of  interest  to  the  training  unit. 

This  can  be  done  easily  if  the  ID  numbers  already  reflect  these  differences 
as  a result  of  Step  2 in  Section  V. 

STEP  3.  Calculate  "Overall  Average  Training  Adequacy"  for  Each  Item. 

The  average  of  all  ratings  of  Adequacy  of  Training  for  Task  should  be 
calculated.  This  can  be  done  rapidly  from  the  separate  trainee  and 
supervisor  averages  in  the  manner  illustrated  in  the  following  example: 

If  there  were  61  trainees  and  the  trainee  average  was  2.79  and 
there  were  79  supervisors  with  an  average  of  2.55,  the  overall  average 
would  be 

(61  x 2.79)  + (79  x 2.55)  = 2.65. 

61  + 79 


Write  this  average  in  the  upper  right-hand  corner  of  the  item  page. 

When  this  average  has  been  calculated  for  all  items,  rearrange  the 
notebook  pages  with  low  values  of  this  average  at  the  beginning  and  high 
values  at  the  end. 

STEP  4.  Calculate  "Average  Training  Adequacy  for  All  Items."  The 
overall  averages  for  each  item,  calculated  in  Step  3 above,  should  be 
summed  and  divided  by  the  total  number  of  items  to  obtain  the  Average 
Training  Adequacy  for  All  Items.  This  provides  a middle  reference  point 
of  Training  Adequacy  to  be  used  to  help  determine  which  of  the  items 
may  require  curriculum  modifications. 
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SECTION  VII 
CURRICULUM  REVISION 

The  completed  analysis  of  data  will  make  it  easy  to  identify  those 
job  tasks  for  which  more  or  less  training  may  be  needed.  After  you 
identify  the  tasks,  you  must  carefully  examine  the  training  requirements 
generated  by  these  items  and  initiate  the  process  of  making  any  needed 
curriculum  changes.  The  following  three  steps  will  help  you  in  this 
task: 


STEP  1.  Identify  Items  with  Low  Average  Ratings  of  Training 
Adequacy  and  Increase  the  Training  Emphasis  as  Needed 

STEP  2.  Identify  Items  with  High  Average  Ratings  of  Training 
Adequacy  and  Reduce  the  Training  Emphasis  where 
Possible 

STEP  3.  Add  New  Items  to  the  Curriculum  when  Recommended  by 
Ten  Percent  or  More  of  the  Respondents 

STEP  1.  Identify  Items  with  Low  Average  Ratings  of  Training  Adequacy 
and  Increase  the  Training  Emphasis  as  Needed.  The  ten  percent  (or  more) 
of  items  with  lowest  average  ratings  of  Training  Adeauacy  require  an 
investigation  of  training  and  possible  curriculum  revisions.  The  items 
with  lowest  average  ratings  may  only  be  one  rating  scale  unit  or  less 
below  the  Average  Training  Adequacy  Rating  for  All  Items.  However,  do 
not  use  this  small  difference  as  an  excuse  to  avoid  curriculum  revision, 
since  many  factors  operate  to  bring  average  ratings  of  Training  Adequacy 
for  an  item  toward  the  midpoint  of  the  rating  scale.  With  150  returned 
questionnaires,  an  item  having  an  average  Training  Adequacy  rating  one- 
half  unit  or  more  below  the  average  Training  Adequacy  rating  calculated 
for  all  items  would  occur  less  than  one  time  in  one  thousand  by  chance. 
Unless  the  school  is  exceptionally  free  of  problems,  the  ten  percent  of 
items  with  lowest  Average  Training  Adequacy  will  be  more  than  one-half 
unit  below  the  Average  Training  Adequacy  for  All  Items.  On  the  initial 
feedback  effort,  more  than  ten  percent  of  the  items  may  take  average 
Training  Adequacy  values  that  are  more  than  one-half  unit  below  the 
Average  Training  Adequacy  for  All  Items.  If  so,  you  will  need  to  consider 
more  than  ten  percent  for  a possible  increase  of  training  emphasis. 

Those  items  with  the  lowest  average  Training  Adequacy  should  receive 
first  attention. 

Since  personnel  at  operational  units  are  aware  of  most  of  the 
factors  that  bear  on  training  adequacy,  you  should  give  much  considera- 
tion to  the  average  rating  of  Training  Adeauacy.  However,  consider 
other  factors  before  revising  the  course.  One  critical  factor  is  the 
rating  of  Task  Frequency  for  the  item.  If  a task  is  performed  by  only  a 


29 


TAEG  Report  No.  20 


very  few  school  graduates,  a Training  Adequacy  rating  which  recommends 
an  increase  in  training  emphasis  would  probably  be  disregarded,  especially 
if  a large  percentage  of  respondents  skipped  the  Training  Adequacy  scale 
for  the  item.  A possible  solution  to  this  training  problem  would  be  a 
special  course  for  this  minority  of  graduates  who  need  increased  training 
for  this  job  task. 

STEP  2.  Identify  Items  with  High  Average  Ratings  of  Training  Adequacy 
and  Reduce  the  Training  Emphasis  where  Possible.  Personnel  at  operational 
units,  especially  supervisors,  often  rate  training  for  job  tasks  as 
somewhat  less  than  adequate.  For  this  reason,  the  Average  Training 
Adequacy  for  All  Items  will  probably  be  less  than  3.0  while  items  with 
highest  Training  Adequacy  ratings  may  only  be  slightly  above  the  3.0 
figure.  Despite  the  actual  numerical  value  of  average  ratings,  the  ten 
percent  (or  more)  of  items  that  are  more  than  .5  units  above  the  Average 
Training  Adequacy  for  All  Items  should  be  considered  as  "overtrained." 

Unless  other  factors  argue  strongly  against  it,  training  emphasis  should 
be  reduced  for  these  items.  Again,  those  items  with  the  most  extreme 
average  ratings  should  receive  first  attention. 

When  these  items  with  highest  average  ratings  of  Training  Adequacy 
also  have  very  low  rates  of  performance,  the  situation  is  probably  one 
of  "irrelevant  training."  More  drastic  curriculum  revisions  are  called 
for  in  these  cases  than  in  the  "overtraining"  condition  where  reduced 
training  is  recommended  (at  least  relative  to  other  items),  but  frequency 
of  performance  is  moderate  or  high. 

Typically,  the  items  at  both  extremes  in  average  Training  Adequacy 
will  have  many  comments  written  about  them.  These  comments  will  aid 
greatly  in  identifying  the  training  problem  and  its  possible  solutions. 

STEP  3.  Add  New  Items  to  the  Curriculum  when  Recommended  by  Ten  Percent 
or  More  of  Respondents.  Job  tasks  recommended  for  addition  to  the 
curriculum  by  ten  percent  or  more  of  respondents  should  be  strongly 
considered  for  inclusion  in  the  revised  curriculum.  Other  items  recommended 
for  inclusion  in  the  curriculum,  but  by  less  than  ten  percent  of  respondents, 
should  be  included  on  the  new  questionnaire  that  will  be  used  in  the 
next  cycle  of  questionnaire  mailings  even  though  the  tasks  are  not 
included  in  the  new  curriculum. 
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SECTION  VIII 

NEW  FEEDBACK  CYCLE:  QUESTIONNAIRE  REVISION  AND  RESPONDENT  SELECTION 

After  the  curriculum  is  modified  and  while  you  wait  for  graduates 
of  the  newly  revised  course  (who  will  provide  the  information  for  the 
next  feedback  cycle)  a number  of  things  need  to  be  done  in  preparation 
for  the  next  feedback  cycle.  These  include  preparation  of  revised  question- 
naires, selection,  and  orientation  of  the  persons  to  receive  them,  and 
maintaining  records  of  feedback  data  from  the  previous  cycle.  Detailed 
procedures  for  these  activities  are  given  in  the  following  five  steps: 

STEP  1.  Identify  and  Correct  Ambiguous  Job  Task  Statements 

STEP  2.  Add  New  Job  Task  Statements  to  Questionnaire 

STEP  3.  Randomly  Select  New  Sample  as  Classes  Graduate  from 
New  Curriculum 

STEP  4.  Provide  Briefing  to  Students  Who  Will  Receive 
Questionnaires 

STEP  5.  Record  Keeping 

STEP  1.  Identify  and  Correct  Ambiguous  Job  Task  Statements.  Despite 
careful  editing  of  job  task  statements  in  section  I,  some  task  statements 
may  not  be  understood  by  all  respondents.  This  will  be  evident  from 
comments  that  are  provided  about  such  items.  It  can  also  be  determined 
from  large  differences  in  average  ratings  of  Frequency  of  Task  between 
trainees  and  supervisors.  Such  differences  often  mean  that  the  inexperienced 
graduates  did  not  correctly  identify  the  equipment  or  some  other  aspect 
of  the  task  statement.  Revise  these  task  statements  to  eliminate  the 
problem  and  then  test  them  on  a number  of  recent  graduates.  If  necessary, 
these  items  should  receive  further  correction  before  including  them  in  the 
new  questionnaire. 

STEP  2.  Add  New  Job  Task  Statements  to  Questionnaire.  Add  new  job 
task  statements  to  the  revised  questionnaire,  whether  they  are  recommended 
by  enough  respondents  to  include  them  in  the  curriculum  or  not.  On  the 
next  round  of  feedback,  these  items  will  receive  Training  Adequacy  ratings 
and  can  be  easily  compared  on  these  ratings  with  the  other  job  task  state- 
ments of  the  questionnaire.  The  more  sensitive  Training  Adequacy  rating  may 
identify  some  job  task  statements  to  be  included  in  the  curriculum  even 
though  they  were  recommended  by  only  a small  percentage  of  respondents 
on  the  previous  questionnaire. 

STEP  3.  Randomly  Select  New  Sample  as  Classes  Graduate  from  New  Curri- 
culum. Using  the  return  rates  obtained  for  supervisors  and  trainees  in 
the  first  round  of  feedback,  calculate  the  number  of  trainees  and 
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supervisors  to  receive  questionnaires  in  order  to  obtain  the  required  75 
returned  questionnaires  from  each  group.  For  example,  if  50  percent  of 
supervisors  responded  and  25  percent  of  trainees  responded,  then  mail 
questionnaires  to  150  supervisors  and  300  trainees  on  the  next  round  of 
feedback.  Calculate  the  percentages  that  these  two  figures  are  of  the 
total  number  of  persons  predicted  to  graduate  from  the  first  six  months 
of  classes  taught  utilizing  the  new  curriculum.  Using  the  randomization 
technique  involving  the  last  two  digits  of  Social  Security  numbers  of 
graduates  (refer  to  Step  3 in  Section  III),  select  the  graduates  who 
will  receive  questionnaires,  and  the  graduates  whose  supervisors  will 
receive  questionnaires,  six  months  following  graduation.  Prepare 
separate  lists  of  these  two  groups  as  was  done  on  the  first  feedback 
effort  (refer  to  Step  5 in  Section  III).  Print  and  assemble  sufficient 
revised  trainee  and  supervisor  questionnaires  to  mail  to  these  two 
groups. 

STEP  4.  Provide  Briefing  to  Students  Who  Will  Receive  Questionnaires. 

Those  students  who  are  selected  to  receive  questionnaires  should  be  con- 
tacted before  they  leave  for  their  new  duty  station  to  make  them  aware 
of  the  future  feedback  function  they  will  perform  and  of  the  importance 
of  this  effort.  This  could  be  done  on  the  last  day  they  are  assembled  as 
a class.  The  questionnaire  can  be  shown  to  the  person  at  this  time,  but 
it  should  not  be  carried  with  him  to  his  new  duty  station  since  past  use 
of  this  technique  indicates  few  questionnaires  would  be  returned  in  six 
months.  Instruct  the  graduate  how  to  report  any  change  of  duty  station 
that  might  occur  within  six  months  of  graduation. 

When  the  last  such  briefing  is  given,  six  months  will  have  passed 
and  the  time  will  have  arrived  to  mail  questionnaires  to  the  first 
graduates  of  the  new  curriculum  or  to  their  supervisors.  Follow  the 
previously  discussed  procedures  for  mailing  the  questionnaires  in  Section 
IV.  The  single  change  to  those  procedures  is  that  you  may  omit  expectation 
letters  for  graduates  who  were  contacted  personally  prior  to  leaving  for 
their  new  duty  stations  (Step  1,  Section  IV). 

STEP  5.  Record  Keeping.  Records  should  be  kept  of  each  feedback  cycle 
for  comparisons  with  future  feedback  cycles.  These  records  should 
include  the  lists  of  persons  receiving  trainee  and  supervisor  questionnaires, 
the  loose-leaf  notebook  with  rearranged  item  pages,  and  single  copies  of 
the  trainee  and  supervisor  questionnaires  with  averages  of  the  respective 
Task  Frequency  and  Training  Adequacy  ratings  included  for  each  item. 

The  Average  Training  Adequacy  for  All  Items  should  also  be  maintained. 

There  is  no  need  to  retain  the  actual  questionnaires  returned  from  the 
operational  environment  once  the  data  have  been  carefully  recorded  in 
the  loose-leaf  notebook.  However,  keep  the  biographical  data  sheet  with 
its  ID  number  in  case  future  analyses  of  the  data  are  required  based  on 
biographical  data  which  were  not  originally  coded  in  the  ID  number  (Step  2, 
Section  V). 
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APPENDIX  A 

GUIDELINES  FOR  DEVELOPING  TASK  STATEMENTS 

Before  attempting  to  develop  task  statements,  there  are  three 
definitions  you  need  to  know.  These  are  (1)  the  definition  of  a job, 

(2)  the  definition  of  a duty,  and  (3)  the  definition  of  a task. 

Job  - A major  classification  based  on  all  duties  and  tasks  actually 
performed.  For  example,  (1)  Sonarman,  (2)  Radioman,  (3) 
Personnel  Specialist,  and  (4)  Quartermaster. 

Duty  - The  major  activities  in  a job.  For  example,  a Radioman's 

duties  include  "process  outgoing  messages,"  "maintain  communica- 
tions message  files,"  and  "restore  malfunctioning  subsystems 
to  normal  operation." 

Task  - Units  of  work  activity  which  make  up  a duty.  A grouping  of 
activities  occurring  close  together  in  time  and  having  a 
common  purpose.  A task  is  a limited  orderly  grouping  of 
purposeful  activities.  Limited  means  small  in  scope  within  a 
convenient  time  period;  orderly  means  having  a logical  and 
observable  beginning  and  end;  purposeful  means  directed 
toward  the  achievement  of  some  objective.  (Examples  are 
shown  below  for  a Personnel  Specialist  and  a Sonarman.) 

Thus,  a job  is  made  up  of  a number  of  duties  which  are  the  major  activities 
in  the  job;  duties  are  made  up  of  a number  of  tasks;  and  each  task  is  an 
activity  composed  of  small  work  units  or  procedural  steps  that  occupy  a 
small  amount  of  time  and  share  a common  goal.  To  illustrate: 
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Example: 

Job:  Personnel  Specialist 

A Duty:  Analyze  and  report  personnel  data 

Task:  Extract  personnel  data  from  records 

Task:  Maintain  status  card  file 

Task:  Prepare  promotion  reports 

Task:  Prepare  quality  control  roster 

Etc. 

Another  Example: 

Job:  Sonar  Operator 

A Duty:  Conduct  sonar  search 

Task:  Determine  predicted  sonar  ranges  for  periscope  depth  and  best 

depth 

Task:  Develop  ray  path  plot 

Task:  Sweep  search  areas  recommended  by  CIC 

Task:  Detect  and  announce  initial  contact 

Task:  Enter  contact  mode 

Task:  Determine  and  announce  contact  classification 

Task:  Track  target  providing  range  and  bearing  information 

Etc. 

Note  from  the  examples  that  each  task  statement  is  a streamlined  but 
complete  sentence;  it  begins  with  a present  tense  action  verb  and  ends 
with  an  object. 

Now  that  you  have  an  idea  of  what  we  mean  by  a "task,"  you  are 
ready  to  examine  the  existing  curriculum  to  determine  the  job  tasks  for 
which  training  is  provided.  Unfortunately,  there  is  no  "set  of  proce- 
dures" or  "cookbook  approach"  that  tells  you  how  to  do  this.  You  must 
rely  on  your  subject  matter  expertise,  your  knowledge  of  the  job,  and 
the  definition  of  a task  provided  above. 
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Since  many  curriculums  are  somewhat  vague,  the  most  difficult  chore 
facing  you  might  be  to  interpret  and  restate  the  goals  of  the  curriculum 
into  the  proper  action  verb-object  form.  Suppose,  for  example,  the 
curriculum  for  an  auto  repair  course  stated,  "The  student  is  required  to 
understand  how  an  internal  combustion  engine  works.  He  will  be  required 
to  demonstrate  his  ability  to  repair  engines."  Obviously,  there  are  a 
host  of  tasks  involved  in  the  repair  of  an  engine.  If  you  were  to 
convert  the  above  statement  into  task  statements,  you  would  include 
statements  such  as: 

(1)  Repair  carburetors 

(2)  Remove  and  replace  defective  spark  plugs 

(3)  Replace  points 

(4)  Set  timing 

(5)  Grind  valves 

(6)  Replace  piston  rings 

You  could  surely  come  up  with  many  more.  It  is  most  important  that  you 
do  identify  all  job  tasks  which  receive  training  emphasis  in  the  curriculum. 
If  you  are  aware  of  other  job  tasks  that  are  not  included  in  the  curriculum, 
these  should  also  be  added  to  your  list,  especially  if  there  is  reason  to 
believe  that  school  training  is  desirable  for  these  tasks. 

Before  writing  your  task  statements,  familiarize  yourself  with  the 
following  helpful  hints  for  writing  task  statements: 

Abbreviations  must  be  used  cautiously  since  they  may  not  be 
understood.  It  is  good  practice  to  spell  out  the  term  and 
follow  it  by  the  abbreviation  in  parentheses  where  it  first 
appears  in  the  inventory.  In  later  tasks,  the  abbreviation 
may  stand  alone. 

Vague  or  ambiguous  words,  such  as  "check,"  "assist,"  "coordinate," 
"recommend,"  "determine,"  "assure,"  should  be  avoided. 

Short  words  should  be  used  in  preference  to  long  words  or 
expressions;  e.g.,  "Write  production  and  control  reports"  NOT 
"Accomplish  necessary  reports  involved  in  the  process  of 
maintaining  production  and  control  procedures." 

. The  task  statement  should  begin  with  a present  tense  action 

word  with  the  subject  "I"  understood;  e.g.,  "operate,"  "write," 
"clean,"  NOT  "operates,"  "writes,"  "cleans." 
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Each  task  statement  must  be  specific  and  capable  of  standing 
alone. 

Each  task  statement  must  be  a complete  sentence.  Do  not  use 
an  action  subheading  followed  by  a series  of  objects;  e.g.. 

Operate  the  following  equipment: 

1.  Automatic  capsule  filler 

2.  Distilling  apparatus 

3.  Force  filters 
Correct  Form 

1.  Operate  automatic  capsule  filler 

2.  Operate  distilling  apparatus 

3.  Operate  force  filters 

Use  "such  as"  followed  by  two  or  three  examples.  Avoid 
"and/or"  and  "etc.";  e.g.,  "requisition  special  items  such  as 
diagnostic  biologicals  and  reagents"  NOT  "requisition  special 
items  such  as  diagnostic  biologicals,  reagents,  etc." 

Use  simple  statements  without  qualifiers,  unless  the  qualifier 
is  essential  to  the  meaning  of  the  statment;  e.g.,  "operate 
power  mower"  NOT  "operate  power  mower  to  cut  grass,"  but 
"schedule  personnel  for  formal  training"  NOT  "schedule  personnel." 

If  a modifier  is  needed  for  greater  specificity,  be  sure  to 
include  all  other  significant  tasks  with  comparable  modifiers. 

For  example,  in  an  automotive  mechanic  inventory,  "repair 
transmissions"  would  probably  be  specific  enough.  However,  if 
the  statement  were  modified  to  read  "repair  automatic  trans- 
missions," then  "repair  standard  transmissions"  should  also  be 
added. 

Avoid  tasks  that  are  obviously  too  specific  or  trivial;  e.g., 
"operate  fork  lift"  NOT  "turn  ignition  key,"  "shift  gears," 
"elevate  fork." 

Avoid  tasks  that  are  too  general.  Such  tasks  will  not 
differentiate  job  tasks;  e.g.,  "repair  carburetors,"  "repair 
body  sections,"  NOT  "repair  motor  vehicles." 
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In  general,  avoid  multiple  verbs  in  a task  statement, 
unless  several  actions  are  invariably  performed  together;  e.g., 
"erect  and  align  poles"  but  NOT  "inspect,  tow,  and  repair 
engines  or  equipment." 

Avoid  redundant  qualifying  phrases  found  in  source  materials 
such  as  "when  appropriate,"  "as  required,"  "in  accordance  with 
prescribed  directives";  e.g.,  "maintain  logs"  NOT  "maintain 
necessary  logs  in  accordance  with  prescribed  Navy-wide  or 
local  regulations  and  directives." 
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APPENDIX  B 

SAMPLE  COVER  LETTER  TO  SUPERVISORS 


39 


SAMPLE  LETTER  TO  SUPERVISOR 


From: 

Curriculum  Update  Division 

School 

To: 

Supervisor  of  Seaman 

Subj  : 

Field  Evaluation  of 

School  training 

Enel: 

(1)  Field  Evaluation  Materials 

1.  As  an  experienced  person  in  your  rating  and  a supervisor 
of  a recent  school  graduate,  you  are  in  an  ideal  position  to 
tell  us  whether  our  graduates  are  meeting  job  requirements 

at  your  unit.  The  enclosed  materials  make  it  possible  for  you 
to  indicate  whether  too  much  or  too  little  emphasis  was  given 
to  any  of  the  various  tasks  covered  in  school.  On  the  final 
page  of  this  questionnaire  we  request  that  you  indicate  job 
tasks  that  are  not  presently  covered  in  school,  but  which 
should  be  covered  in  the  future.  Throughout  your  completion 
of  these  materials,  we  hope  you  will  write  down  any  thoughts 
you  may  have  about  training  problems,  recommendations  for 
their  solution,  and  any  other  aspects  of  school  training. 

2.  Please  return  these  materials  in  the  enclosed  envelope 
within  two  weeks,  if  possible.  This  information  will  aid  us 

to  provide  better  training  of  

in  the  future. 

3.  If  you  have  recently  completed  field  evaluation  materials 

for  School  there  is  no  need  to  complete 

these  unless  you  have  some  additional  recommendations.  However, 
we  would  appreciate  if  you  would  pass  these  materials  on  to 

some  other  experienced  who  is 

familiar  with  the  above  person's  work. 


CHIEF  PETTY  OFFICER 
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APPENDIX  C 

SAMPLE  COVER  LETTER  TO  TRAINEE 
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SAMPLE  LETTER  TO  TRAINEE 


From: 

Curriculum  Update  Division 

School 

To: 

Seaman 

Sub  j : 

Field  Evaluation  of 

School 

Training 

Enel: 

(1)  Field  Evaluation  Materials 

1.  At 

one  of  your  last  classes  in  school 

the  important  task 

you  can 

perform  in  the  identification  of  training  problems  was  discussed.  At 
this  time,  we  are  asking  you  to  aid  us  in  this  task  since  you  have 
probably  been  on  the  job  long  enough  to  have  developed  a good  under- 
standing of  your  duties  and  the  training  needed  to  perform  them. 

2.  The  enclosed  materials  make  it  possible  for  you  to  indicate 
whether  too  much  or  too  little  emphasis  was  given  to  any  of  the 
various  tasks  covered  in  school.  On  the  final  page  of  this  questionnaire 
we  request  that  you  indicate  job  tasks  that  are  not  presently  covered  in 
school,  but  which  should  be  covered  in  the  future.  Throughout  your 
completion  of  these  materials  we  hope  you  will  write  down  any  thoughts 
you  may  have  about  training  problems,  recommendations  for  their  solution, 
and  any  other  aspects  of  school  training. 

3.  Please  return  these  materials  in  the  enclosed  envelope  within  two 

weeks,  if  possible.  This  information  will  aid  us  to  provide  better 
training  of  in  the  future. 


Chief  Petty  Officer 
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PLEASE  FURNISH  THE  FOLLOWING  INFORMATION  ABOUT  YOURSELF  AND  THE  RECENT  RADIOMAN  "A" 
SCHOOL  GRADUATE  WHOM  YOU  SUPERVISE 


1. 

Your 

name 

2. 

Rank 

3.  Today's 

Date 

4. 

Duty 

Station 

5. 

Name  of 

recent  RM  "A"  School 

graduate 

6.  His  rank 7.  How  many  months  has  he  been 

assigned  to  his  present  duty  station? 8.  Has  he  been  assigned  to  duty  within 

his  job  specialty? If  No,  what  is  his  assignment? 

9.  Did  he  attend  International  Morse  Code  School  or  any  other  school  after  completing 

Radioman  "A"  School? If  Yes,  which  school? __ 

10.  Did  mess-cooking  or  other  activities  at  this  duty  station  delay  the  beginning  of  his 

work  in  his  specialty? If  Yes,  what  activities? 

How  long  was  the  delay? 11.  Check  the  following  statement  that  best 

describes  how  much  this  man's  RM  "A"  School  training  is  utilized  in  his  present  job? 


VERY  MUCH 


MUCH 


SOME 


VERY  LITTLE 


NOT  AT  ALL 
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PLEASE  FURNISH  THE  FOLLOWING  INFORMATION 


1. 

Name 

2. 

Rank 

3.  SSN 

4. 

Today ' s date 

5. 

Present 

Duty 

Station 

6. 

How  many  months 

have  you  been  at  your 

present 

duty 

station? 

Date  arrived 7.  Have  you  been  assigned  to  duty  within  your  job 

specialty? If  No,  what  is  your  assignment? 

8.  Did  Mess-Cooking  or  other  activities  at  this  duty  station  delay  the  beginning  of 

work  in  your  specialty? If  Yes,  what  activities? 

_ How  long  was  the  delay? 

9.  Did  you  attend  International  Morse  Code  School  or  any  other  school  after  Radioman 

"A"  School? If  Yes,  which  school? 

10.  Check  the  following  statement  that  best  describes  how  much  the  training  you  received 
at  the  Radioman  "A"  School  is  used  in  your  present  job. 

VERY  MUCH  MUCH  SOME 


VERY  LITTLE 


NOT  AT  ALL 
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INSTRUCTIONS  TO  SUPERVISOR  FOR  COMPLETING  RATING  SCALES 


On  the  following  pages  tasks  are  listed  which  received  at  least  some  emphasis  in 
school.  Please  rate  each  task  on  the  two  scales  at  the  right  of  the  task  by  circling 
the  most  appropriate  number.  On  the  "Frequency  of  Task"  scale,  select  the  category 
that  corresponds  most  closely  to  the  actual  frequency  with  which  this  task  is  performed 
by  the  recent  school  graduate  that  you  supervise  in  his  present  assignment. 

On  the  "Adequacy  of  School  Training  for  This  Task"  scale,  select  the  most  appro- 
priate of  the  following  categories: 

1.  Task  requires  much  more  emphasis  in  school. 

2.  Training  less  than  adequate  for  task,  increase  emphasis. 

3.  Training  adequate  for  task. 

4.  Training  more  than  adequate  for  task,  reduce  emphasis. 

5.  Greatly  reduce  or  eliminate  training  for  this  task. 

(You  may  skip  this  "Adequacy"  rating  for  a particular  task  if  that  task  is  never 
performed  and  you  do  not  feel  you  can  rate  adequacy  of  training  for  it.) 

In  making  this  rating  consider  such  things  as  the  following: 

Problems  he  may  have  had  performing  this  task  when  first  required  to  do  it; 
The  amount  of  time  that  was  required  by  you  or  by  others  at  your  unit  to 


bring  him  "up-to-speed" ; 
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Whether,  for  some  reason,  the  task  should  have  been  learned  on-the-job 


instead  of  in  school;  and 

Whether  learning  to  perform  this  task  in  school  does  not  help  this  man  in 
his  present  job  or  will  not  help  him  in  the  foreseeable  future. 

Your  experience  in  your  rating  makes  you  uniquely  qualified  to  judge  when  job 
tasks  need  more  or  less  school  emphasis.  Not  only  have  you  already  considered  the 
question  of  what  is  the  proper  balance  between  school  training  and  training  on  the 
job,  but  you  can  also  see  the  possible  future  value  of  training  that  has  little 
immediate  use.  We  look  forward  to  seeing  your  ratings  of  training  adequacy  and  will 
give  them  much  consideration. 
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INSTRUCTIONS  TO  TRAINEE  FOR  COMPLETING  RATING  SCALES 


On  the  following  pages  tasks  are  listed  which  received  at  least  some  emphasis  in 
school.  Please  rate  each  task  on  the  two  scales  at  the  right  of  the  task  by  circling 
the  most  appropriate  number.  On  the  "Frequency  of  Task"  scale,  select  the  category 
that  corresponds  most  closely  to  the  actual  frequency  with  which  this  task  is  performed 
by  you  in  your  present  assignment. 

On  the  "Adequacy  of  School  Training  for  This  Task"  scale,  select  the  most  appro- 
priate of  the  following  categories: 

1.  Task  requires  much  more  emphasis  in  school. 

2.  Training  less  than  adequate  for  task,  increase  emphasis. 

3.  Training  adequate  for  task. 

4.  Training  more  than  adequate  for  task,  reduce  emphasis. 

5.  Greatly  reduce  or  eliminate  training  for  this  task. 

(You  may  skip  this  "Adequacy"  rating  for  a particular  task  if  that  task  is  never 
performed  and  you  do  not  feel  you  can  rate  adequacy  of  training  for  it.) 

In  making  this  rating  consider  such  things  as  the  following: 

Problems  you  may  have  had  performing  this  task  when  first  required  to  do  it; 
The  amount  of  time  that  was  required  by  your  supervisor  or  others  at  your 
unit  to  bring  you  "up-to-speed"  on  the  task; 
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Whether,  for  some  reason,  the  task  should  have  been  learned  on-the-job 
instead  of  in  school;  and 

Whether  learning  to  perform  this  task  in  school  does  not  help  you  in  your 
present  job  or  will  not  help  you  in  the  foreseeable  future. 

Also  consider  that  school  training  is  expensive  and  must  be  used  only  for 
essential  tasks.  On  the  other  hand,  remember  that  operational  units  have  many  other 
functions  to  perform  beside  on-the-job  training. 

As  you  can  see,  the  rating  of  training  adequacy  is  not  simple.  We  are  asking 
you  to  do  this  since  you  hold  two  views  of  the  world  that  are  critical  for  judging  the 
adequacy  of  training.  One  view  is  of  school  training  as  it  exists  for  the  student  and 
the  other  view  is  of  the  requirements  of  your  present  job.  These  unique  perspectives 
of  yours  make  your  careful  ratings  invaluable  to  us! 


r 
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Although  we  have  already  asked  you  to  consider  existing  school  training  in  great 
detail,  there  is  one  more  very  important  job  you  can  do  for  us.  We  need  to  know  what 


things  presently  are  NOT  taught  in  school  but  should  be  taught  there.  Consider  things 
the  trainee  has  had  to  learn  on  the  job  with  much  loss  of  time  for  both  him  and  his 
supervisors.  Also  consider  tasks  he  still  cannot  perform  because  he  did  not  learn  them 
in  school  and  because  it  has  not  been  possible  to  train  him  on  the  job.  Please  do  this 
carefully  and  thoughtfully.  As  a supervisor  of  a recent  school  graduate,  you  are  in  a 
unique  position  to  identify  those  things  which  are  almost  certain  to  be  missing  from 
school . 

cn 

o> 

1. 


2. 


3. 
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Although  we  have  already  asked  you  to  consider  existing  school  training  in  great 


detail,  there  is  one  more  very  important  job  you  can  do  for  us.  We  need  to  know  what 
things  presently  are  NOT  taught  in  school  but  should  be  taught  there.  Consider  things 
you  have  had  to  learn  on  the  job  with  much  loss  of  time  for  both  you  and  your  super- 
visors. Also  consider  tasks  you  still  cannot  perform  because  you  did  not  learn  them  in 
school  and  because  it  has  not  been  possible  to  train  you  on  the  job.  Please  do  this 
carefully  and  thoughtfully.  As  a school  graduate  working  in  the  job  you  were  trained 
to  do,  you  are  in  a unique  position  to  identify  those  things  which  are  almost  certain 
to  be  missing  from  school. 

1.  

cn 

Co 


2. 


3. 


4. 
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From:  Curriculum  Update  Branch,  School 

To:  Seaman  

In  two  weeks  you  will  receive  a questionnaire  which  will  enable 

you  to  indicate  whether  the  training  you  received  in  

School  meets  the  needs  of  your  new  job.  Specific  job  tasks  will  be 
mentioned  on  this  form  and  you  will  be  asked  to  indicate  how  often  you 
perform  the  task  and  whether  more  or  less  school  training  should  have 
been  provided  for  the  task.  Please  spend  some  time  thinking  about  the 
adequacy  of  your  school  training  as  you  go  about  your  duties  during  the 
next  two  weeks.  One  area  of  particular  importance  are  job  tasks  which 
should  have  been  included  in  school  but  were  not.  A special  section  of 
the  questionnaire  will  provide  space  for  any  such  items. 

With  your  assistance  the  problems  of School 

can  be  identified  and  the  necessary  steps  taken  to  correct  them.  Please 
complete  and  return  the  questionnaire  promptly  when  it  arrives. 


Chief  Petty  Officer 
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From:  Curriculum  Update  Branch,  _ School 

To:  Recent School  Graduate 

About  one  month  ago  you  were  mailed  materials  for  evaluating  the 

training  at School.  It  is  most  important  that  we 

receive  your  completed  questionnaire  in  order  that  we  may  use  the  data 

in  our  continuing  program  to  make  training  of  _ 

both  relevant  and  effective. 

If  you  did  not  receive  the  materials  or  the  materials  were  incom- 
plete, please  contact  us  and  we  will  mail  another  set.  If  you  did 
receive  them,  please  complete  and  return  them  as  soon  as  possible. 

Sincerely, 

Chief  Petty  Officer  
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DISTRIBUTION  LIST 


Air  Force 


Military  Assistant  for  Human  Resources,  OAD  (E&LS)  ODDR&E  (LTCOL  Henry  Taylor) 
Headquarters,  Air  Training  Command  (XPTD,  Dr.  D.  E.  Meyer) 

Air  Force  Human  Resources  Laboratory,  Brooks  Air  Force  Base 

Air  Force  Human  Resources  Laboratory  (Library),  Lowry  Air  Force  Base 

Air  Force  Office  of  Scientific  Research  (Dr.  A.  R.  Fregly) 

Army 

Army  Research  Institute  (Dr.  Ralph  R.  Canter,  316C) 

President,  Combat  Arms  Training  Board  (TB-ILD) 

Headquarters,  TRADOC  (ATTNG:EA) 

Coast  Guard 


U.S.  Coast  Guard  Headquarters  (G-P-l/62) 

Marine  Corps 

CG  MCDEC  (Mr.  Greenup) 

Director,  Marine  Corps  Institute 

Navy 

Assistant  Secretary  of  the  Navy  (R&D)  (4E741 , Dr.  S.  Koslov) 

CNO  (0P-987P7,  CAPT  H.  J.  Connery;  OP-991 B,  M.  Malehorn;  0P-987P10, 

Dr.  R.  Smith;  OP-987,  H.  Stone) 

COMNAVELEX  (Code  03) 

COMNAVSEASYSCOM  (Code  03,  047C1 , 047C12) 

COMNAVAIRSYSCOM  (Code  03) 

CNM  (MAT-03424,  Mr.  A.  L.  Rubinstein) 

COMNAVAIRSYSCOM  (Code  340F) 

ONR  (Code  458,  455) 

ONRBO  Boston  (J.  Lester) 

ONRBO  Chicago 

ONRBO  Pasadena  (E.  E.  Gloye) 

Strategic  Systems  Project  Office  (Mr.  J.  F.  Grosson) 

CNET  (00A,  N-5  (6  copies),  N-5A) 

CNET  Liaison  (CDR  Max  Quitiquit,  AFHRL/FTLNN) , Williams  Air  Force  Base 
CO  NAVEDTRAPRODEVCEN  (AH3,  Dr.  E.  E.  Lewis) 

CNETS  (N-2  (4  copies).  Library) 

CNTECHTRA  (0161,  Dr.  Kerr;  Dr.  K.  Johnson;  Library) 

CNATRA  (301,  F.  Schufletowski ) 

CNAVRES  (Code  02) 

COMTRALANT 

COMTRALANT  (Educational  Advisor) 

COMTRAPAC 


DISTRIBUTION  LIST  (continued) 


CO  NAVEDTRASUPPCEN  NORVA 
CO  NAVEDTRASUPPCENPAC 

CO  NAVPERSRANDCEN  (CODE  02,  Dr.  Regan  (5  copies)) 

NAVPERSRANDCEN  Liaison  (Code  01H) 

NAMRL  (Chief  Aviation  Psych.  Div.) 

NETISA  (Code  00) 

CO  NAVSUB  Base  NLON  (Psychology  Section) 

COMNSWC 

U.  S.  Naval  Institute  (CDR  Bowler) 

NAVPGSCOL  (Code  2124) 

U.  S.  Naval  Academy  (Chairman,  Behavioral  Science  Dept.) 

CO  NAVTRAEQUIPCEN  (N-215,  N-22,  N-131  (2  copies),  N-017,  N-OOAF,  N-OOM, 
N-OOA) 

Other  DoD 


Director,  Human  Resources  Office,  ARPA  (R.  Young) 

Defense  Advanced  Research  Projects  Agency  (Dr.  H.  F.  O'Neill,  Jr.) 

Information  Exchanges 

DDC  (12  copies) 

DLSIE  (James  Dowling) 

Scientific  Technical  Information  Office,  NASA 

Executive  Editor,  Psychological  Abstracts,  Amercian  Psychological  Association 
ERIC  Processing  and  Reference  Facility,  Bethesda,  MD.  (2  copies) 
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DISTRIBUTION  LIST  (continued) 


CNATRA  (3  copies) 

CNTECHTRA  (3  copies) 

CNETS  (3  copies) 

COMTRAPAC  (3  copies) 

COMTRALANT  (3  copies) 

CO  NAS  Chase  Field 
CO  NAS  Corpus  Christi 
CO  NAS  Kingsville 
CO  NAS  Meridian 
CO  NAS  Pensacola  (5  copies) 

CO  NAS  Saufley  Field,  Pensacola 
CO  NAS  Whiting  Field,  Milton 
CO  NAVAVSCOLSCOM  (3  copies) 

OIC  CBU-407  (2  copies) 
COMTRAWING  ONE 
COMTRAWING  TWO 
COMTRAWING  THREE 
COMTRAWING  FOUR 
COMTRAWING  FIVE 
COMTRAWING  SIX 
CO  TRARON  ONE 
CO  TRARON  TWO 
CO  TRARON  THREE 
CO  TRARON  FOUR 
CO  TRARON  FIVE 
CO  TRARON  SIX 
CO  TRARON  SEVEN 
CO  HELTRARON  EIGHT 
CO  TRARON  NINE 
CO  TRARON  TEN 
CO  HELTRARON  EIGHTEEN 
CO  TRARON  NINETEEN 
CO  TRARON  TWENTY-ONE 
CO  TRARON  TWENTY-TWO 
CO  TRARON  TWENTY-THREE 
CO  TRARON  TWENTY- FOUR 
CO  TRARON  TWENTY- FIVE 
CO  TRARON  TWENTY-SIX 
CO  TRARON  TWENTY -SEVEN 
CO  TRARON  TWENTY-EIGHT 
CO  TRARON  TWENTY-NINE 
CO  TRARON  THIRTY-ONE 
CO  TRARON  EIGHTY-SIX 
CO  NFDS 

CO  NAS  Memphis 

CO  NTTC  Meridian  (5  copies) 

CO  NATTC  Lakehurst  (3  copies) 

CO  NATTC  Memphis  (5  copies) 
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DISTRIBUTION  LIST  (continued) 


CO  NAMTRAGRU  (3  copies) 

CO  NAVU  Lowry  AFB 
OIC  CBU-404  (2  copies) 

CO  NAVCRUITRACOM  Great  Lakes  (5  copies) 
CO  NAVCRUITRACOM  Orlando  (5  copies) 

CO  NAVCRUITRACOM  San  Diego  ( 5 copies) 

CO  SERVSCOLCOM  Great  Lakes  (16  copies) 

CO  SERVSCOLCOM  Orlando  (5  copies) 

CO  SERVSCOLCOM  San  Diego  (12  copies) 

CO  NAVSCOLCOM  Treasure  Island  (5  copies) 
CO  COMBATSYSTECHSCOLSCOM  (5  copies) 

CO  FLEBALMISUBTRACEN  (3  copies) 

COMNETC  Newport  (11  copies) 

COMNTC  Bainbridge 

COMNTC  Great  Lakes  (5  copies) 

COMNTC  Orlando  (5  copies) 

COMNTC  San  Diego  (5  copies) 

CO  NAVDAMCONTRACEN  (3  copies) 

CO  NAVSUBTRACENPAC  (3  copies) 

CO  NTTC  Pensacola  (3  copies) 

OIC  NAVADMINU  Idaho  Falls 
CO  NAVADMINU  Scotia 
CO  NAVADMINCOM  Great  Lakes 
CO  NAVADMINCOM  San  Diego 
CO  NAVPHIBSCOL  Coronado  (3  copies) 

CO  NAVPHIBSCOL  Little  Creek  (3  copies) 

CO  NAVSCOL  BT  Philadelphia  (3  copies) 

CO  NAVSCOLCECOFF  Port  Hueneme  (3  copies) 
OIC  COMSYSTECHSCOL  Washington  (3  copies) 
CO  NAVSCOLCONST  Davisville  (3  copies) 

CO  NAVCONSTRACEN  Port  Hueneme  (3  copies) 
CO  NAVCONSTRACEN  Gulfport  (3  copies) 

CO  NAVDESCOL  Newport  (3  copies) 

CO  NAVSCOLDIVESALV  Washington  (3  copies) 
CO  NAVSCOLEOD  Indian  Head  (3  copies) 

CO  NAVGMSCOL  Dam  Neck  (3  copies) 

CO  NAVJUSTSCOL  Newport  (3  copies) 

CO  NAVSUBSCOL  Groton  (3  copies) 

CO  NAVSCSCOL  Athens  (3  copies) 

CO  NAVSCOLTRANSMGT  Oakland  (3  copies) 

OIC  CTS  for  PTP  Dam  Neck  (3  copies) 

CO  NAVNUPWRSCOL  Bainbridge  (3  copies) 

CO  NAVNUPWRSCOL  Mare  Island  (3  copies) 

CO  NAVNUPWRTRAU  Idaho  Falls  (3  copies) 

CO  NAVNUPWRTRAU  Schenectady  (3  copies) 

CO  NAVNUPWRTRAU  Windsor  (3  copies) 

CO  FLETRACEN  San  Diego  (5  copies) 
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DISTRIBUTION  LIST  (continued) 

CO  FLTCOMBATDIRSYSTRACENPAC  (3  copies) 

CO  FLEASWTRACENPAC  (3  copies) 

CO  FITCPAC  (3  copies) 

CO  FLETRACEN  Mayport  (5  copies) 

CO  FLETRACEN  Norfolk  (5  copies) 

CO  FLEMINEWARTRACEN  (5  copies) 

CO  FLESONARSCOL  (3  copies) 

CO  FL EASWTRAC EN LANT  (3  copies) 

CO  FLTCOMBATDI RSYSTRACENLANT  (3  copies) 

CO  NAVSCOLCRYOGENICS  (3  copies) 

CO  FITCLANT  (3  copies) 

CO  School  of  Music  Norfolk  (3  copies) 

CO  MATSG  Pensacola 
CO  MATSG  Corpus  Christi 
CO  MATSG  Memphis 
CO  MATSG  Lakehurst 

CO  NAVEDTRASUPPCEN  Atlantic  (10  copies) 

CO  NAVEDTRASUPPCEN  Pacific  (6  copies) 

CO  NAVTRAEQUIPCEN  (2  copies) 

CO  NAVEDTRAPRODEVCEN  (3  copies) 

DIR  NAVINSTECHDEVCEN  (2  copies) 

PRES  NAVWARCOL  (3  copies) 

SUPT  NAVPGSCOL  (3  copies) 

CO  NAVADMINCOM  Norfolk  (3  copies) 

DIR  DEFREMANEDCEN  (3  copies) 

SUPT  USNA  ANNA  (3  copies) 

SUPT  DEPNEDOFLANT  (3  copies) 

OIC  CBU-403  (2  copies) 

DIR  MARDAC 
CO  NETISA 
DIR  DANTES 
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